Methods, systems and apparatuses for high-quality green imbalance compensation in images

ABSTRACT

Methods, systems and apparatuses for determining and compensating for green imbalance in imagers. An estimated local green balance correction term is limited, e.g., between an upper and lower limit. The upper and lower limits are determined from a set of parameters. The parameters are determined during a calibration procedure. The parameters may be stored for a subset of possible pixel locations and parameters for each pixel not at a location corresponding to stored data may be determined by interpolation.

FIELD OF THE INVENTION

Embodiments of the invention relate to methods, systems and apparatusesfor green imbalance compensation in imagers.

BACKGROUND OF THE INVENTION

Imagers, such as for example charge coupled devices (CCD), complementarymetal oxide semiconductor (CMOS) and others, are widely used in imagingapplications including digital still and video cameras. A CMOS imagercircuit includes a focal plane array of pixels, each one of the pixelsincluding a photosensor, for example, a photogate, a photoconductor, aphototransistor or a photodiode for accumulating photo-generated chargein the specified portion of the substrate. Each pixel has a chargestorage region, formed on or in the substrate, which is connected to thegate of an output transistor that is part of a readout circuit. Thecharge storage region may be constructed as a floating diffusion region.In some imager circuits, each pixel may include at least one electronicdevice such as a transistor for transferring charge from the photosensorto the storage region and one device, also typically a transistor, forresetting the storage region to a predetermined charge level prior tocharge transference.

In a CMOS imager, the active elements of a pixel perform the functionsof: (1) photon to charge conversion; (2) accumulation of image charge;(3) resetting the storage region to a known state; (4) transfer ofcharge to the storage region; (5) selection of a pixel for readout; and(6) output and amplification of signals representing pixel reset leveland pixel charge. Photo charge may be amplified when it moves from theinitial charge accumulation region to the storage region. The charge atthe storage region is typically converted to a pixel output voltage by asource follower output transistor.

Exemplary CMOS imaging circuits, processing steps thereof, and detaileddescriptions of the functions of various CMOS elements of an imagingcircuit are described, for example, in U.S. Pat. Nos. 6,140,630;6,204,524; 6,310,366; 6,326,652; 6,333,205; 6,376,868; 6,852,591, all ofwhich are assigned to Micron Technology, Inc. The disclosures of each ofthe foregoing are hereby incorporated by reference herein in theirentirety.

CMOS imagers typically have an array of pixels containing photosensors,where each pixel produces a signal corresponding to the intensity oflight impinging on that element when an image is focused on the pixelarray. The signals may then be digitized and stored, for example, fordisplay of a corresponding image on a monitor or for providing hardcopyimages or otherwise used to provide information about a captured image.The magnitude of the signal produced by each pixel is proportional tothe intensity of light impinging on the respective photosensor.

For photosensors to capture a color image, they must be able toseparately detect color components of a captured image. For example whenusing a Bayer pattern, as shown for example in FIG. 1, photons having awavelength corresponding to a red, green or blue color are detected byrespective red, green, and blue pixels (i.e., each pixel is sensitiveonly to one color or spectral band). For this to occur, a color filterarray (CFA) is typically placed in front of the pixel array so that eachpixel receives the light of the color of its associated filter accordingto a specific pattern, e.g., the Bayer pattern 10 of FIG. 1. Other colorfilter array patterns are also known in the art and are applicable aswell.

As shown in FIG. 1, the Bayer pattern 10 is an array of repeating red(R), green (G), and blue (B) filters. A red pixel is a pixel covered bya red filter; similarly, a blue pixel or a green pixel is a pixelcovered by a blue or a green filter, respectively. The pixels of FIG. 1may be identified by coordinates, p_(i,j), to identify the color and thelocation of the pixel within the pixel array, where p indicates thecolor (R for red, B for blue, G for green), i indicates the rowlocation, and j indicates the column location. For example, one row 15includes a green pixel G_(1,1) in column one and a red pixel R_(1,2) incolumn two. Likewise, a next row 20 includes a blue pixel B_(2,1) incolumn one and a green pixel G_(2,2) in column two.

In the Bayer pattern 10, red pixels 11, green pixels 13 and blue pixels12 are arranged so that alternating red 11 and green 13 pixels are inone row 15 of a pixel array, and alternating blue 12 and green 13 pixelsare in a next row 20. These alternating rows 15, 20 are repeatedthroughout the pixel array. Thus, when the imager is read out, the pixelsequence for one row (i.e., row 15) reads GRGRGR, etc., and the sequencefor the next row (i.e., row 20) reads BGBGBG, etc. While FIG. 1 depictsan array having only ten rows and ten columns, pixel arrays typicallyhave hundreds or thousands of rows and columns of pixels.

Pixels of a pixel array may experience interference, or crosstalk, whenneighboring pixels interfere with each other. One form of suchinterference is optical crosstalk. Other forms of such interferenceinclude electrical crosstalk. Crosstalk may cause different problems ina captured image, one of which is a phenomenon known as green imbalance,which is when the pixels in one green channel of a pixel array provide adifferent output signal than the pixels in the other green channel ofthe pixel array under the same illumination level.

An example of optical crosstalk is described with reference to FIG. 2.FIG. 2 illustrates a portion of a color filter array 35 having the Bayerpattern 10 in front of a portion of pixels 40 of a pixel array. Forpixels 40 located, for example, on the periphery of the pixel array,light rays 30 from a lens, e.g. a camera lens, may be coming in atoblique angles. In FIG. 2, for example, the illumination in the redspectrum which passes through red filter 36 at an oblique anglesubstantially affects the response of the green pixel 43. This isbecause the light rays 30 are coming in at such an angle that the lightpassing through red filter 36 actually hits the green pixel 43 insteadof the red pixel for which it is intended. The same type of effect wouldalso occur for green pixels that are located adjacent to blue pixels.The magnitude of the affect of optical crosstalk on a specific pixel isa function of several factors, including, for example, the distancebetween the pixel and its neighboring pixels and the distance between aphoto sensor and an overlying microlens. Optical crosstalk may affectpixels throughout the pixel array and is not limited in its effect topixels located at the periphery. However, the magnitude of theinterference will depend on the angle of incidence of the light rays andtherefore will vary throughout a pixel array depending on the locationof the pixel within the pixel array.

Electrical crosstalk occurs when electrical charge generated in thephoto sensor of one pixel travels to be collected at or otherwiseinfluences the signal of an adjacent pixel. Such crosstalk causesinterferences between signals converted via the photo detecting device.

The crosstalk interference of the red and blue pixels with the greenneighboring pixels may cause the green-red pixels and the green-bluepixels to appear differently, even in response to the same level oflight stimulus. This occurrence is known as green imbalance. Green-redpixels are the green pixels which appear in the same rows as red pixels,e.g., row 15 of FIG. 1. Green-blue pixels are the green pixels whichappear in the same rows as blue pixels, e.g., row 20 of FIG. 1. Thegreed-red and green-blue pixels appear differently because the bluepixels will have a different affect on the green-blue pixels than thered pixels will have on the green-red pixels. In turn, this causes thepixels of the two green channels to have different crosstalk effects.

The presence of green imbalance can degrade image quality unless specialsteps are taken to counteract that effect during image processing. Animage captured of an evenly illuminated white field, for example, mayresult in responses of the green-red pixels being different than thegreen-blue pixels. If this effect is not corrected during imageprocessing, the variation in response may show up as a fine checkerboardpattern overlaid on a captured image or as other forms of imageartifacts. Therefore, this imbalance between the green-blue and thegreen-red pixels should be desirably corrected during image processing.Ideally, the crosstalk components of the green-blue and green-red pixelsignals are normalized after green imbalance compensation, at whichpoint any effect of crosstalk will no longer be apparent in thedisplayed image.

Various computational methods are known to exist to compensate for greenimbalance, including methods utilizing local neighborhood operations.Existing local green balancing methods improve image quality, but mayalso yield abundant undesirable image artifacts. Accordingly, a methodthat delivers high-quality results for green imbalance compensationwithout adding undesirable artifacts is desired.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 shows a Bayer color filter pattern.

FIG. 2 shows an example of optical crosstalk.

FIG. 3 is a graph showing pixel response versus correction term limitsfor green imbalance compensation.

FIG. 4 is a flowchart showing a method of imager calibration inaccordance with disclosed embodiments.

FIG. 5 is a flowchart showing an additional method of imager calibrationin accordance with disclosed embodiments.

FIG. 6 is a flowchart showing a method for performing green imbalancecompensation in accordance with disclosed embodiments.

FIG. 7 is a flowchart showing a method for determining correction termlimits in accordance with disclosed embodiments.

FIG. 8 is a block diagram of a limits calculator of disclosedembodiments.

FIG. 9 is a block diagram of a system-on-a-chip imager implementingdisclosed embodiments.

FIG. 10 is an example of a sensor core used in the FIG. 9 imager.

FIG. 11 is a processing system, for example, a digital still or videocamera system, constructed in accordance with disclosed embodiments.

DETAILED DESCRIPTION OF THE INVENTION

In the following detailed description, reference is made to theaccompanying drawings which form a part hereof, and in which are shownby way of illustration specific embodiments of the invention. Theseembodiments are described in sufficient detail to enable those skilledin the art to make and use them, and it is to be understood thatstructural, logical or procedural changes may be made. Particularly, inthe description below, processing is described by way of flowchart. Insome instances, steps which follow other steps may be in reverse, in adifferent sequence, or may occur simultaneously, except where afollowing procedural step requires the presence of a prior proceduralstep. The processes illustrated in the flowcharts can be implemented asa pixel processing pipeline circuit provided in an image processor of asolid state imager device. The pixel processing pipeline circuit can beimplemented using hardware components including an ASIC, a processorexecuting a program, or other signal processing hardware and/orprocessor structures or any combination thereof.

In the following description, the embodiments are described in relationto a CMOS image sensor for convenience purposes only; the disclosedembodiments, however, have wider applicability to green imbalancecompensation for pixel output signals in any image sensor, including CCDimagers.

As noted, conventional technologies are known for compensating for greenimbalance in an imager. As previously described, FIG. 1 shows a portionof a pixel array having a Bayer color filter pattern 10. In general,during green imbalance compensation in one known technique, for eachpixel, p_(i,j), in the array that has a green color filter (e.g.,G_(ij)), local green balancing algorithms calculate a local greenbalance correction term, Δ_(i,j). The green balance correction term maytake into account optical and/or electrical crosstalk. The green pixelsignals are adjusted by applying this green balance correction term tothe pixel response signal, thereby compensating for the crosstalk effectand correcting a captured image. One example of an implementation of alocal green balancing method is disclosed in copending application Ser.No. 11/783,865, entitled METHOD, APPARATUS AND SYSTEM PROVIDINGGREEN-GREEN IMBALANCE COMPENSATION, filed Apr. 12, 2007 (“the '865application”) the entire disclosure of which is incorporated herein byreference.

One possible method for calculating the local green balance correctionterm, Δ_(i,j), for a particular pixel, p_(i,j), is now described. Thismethod is described in more detail in the '865 application. In general,local green balancing methods determine a correction term based on anaverage of the pixel signal values for the surrounding green pixels. Thesurrounding green pixels may be in the form of a pixel kernel 25 of sizeM by N, shown as a 5×5 pixel kernel, for example, in FIG. 1. Thecorrection term for a pixel in the center of pixel kernel 25 (e.g.,G_(3,3) in FIG. 1) is calculated as shown in Equations (1) through (3):

$\begin{matrix}{{S_{G\; 1} = \frac{G_{1,1} + G_{1,3} + G_{1,5} + G_{3,1} + G_{3,3} + G_{3,5} + G_{5,1} + G_{5,3} + G_{5,5}}{9}};} & (1) \\{{S_{G\; 2} = \frac{G_{2,2} + G_{2,4} + G_{4,2} + G_{4,4}}{4}};{and}} & (2) \\{\Delta_{i,j} = {\frac{S_{G\; 1} - S_{G\; 2}}{2}.}} & (3)\end{matrix}$As can be seen in Equations (1) through (3), two averages are firstcalculated. S_(G1) is the average of pixel signals (G_(i,j)) that arelocal (e.g., within the same M by N pixel kernel) and belong to the samegreen color channel as the pixel being corrected (e.g., all green-redpixels in FIG. 1). S_(G2) is the average of pixel signals that are localand belong to the other green color channel as the pixel being corrected(e.g., all green-blue pixels in FIG. 1).

The corrected pixel signal for a given green pixel located at the centerof the pixel kernel 25 may then be calculated as in one of Equations(4), (5) or (6), depending on the relationship between the twocalculated averages, S_(G1) and S_(G2):If S _(G1) >S _(G2), then p′ _(i,j) =p _(i,j)−|Δ_(i,j)|; or   (4)If S _(G1) <S _(G2), then p′ _(i,j) =p _(i,j)+|Δ_(i,j)|; or   (5)If, S _(G1) =S _(G2), then p′ _(i,j) =p _(i,j),   (6)where p′_(i,j) represents the corrected pixel signal, p_(i,j) representsthe pixel signal from the imager pixel array and Δ_(i,j) is thecorrection term calculated in accordance with Equation (3). Thecorrection term may be calculated in this manner for each pixel in thepixel array using the corresponding pixel kernel of which the pixelbeing corrected is at the center, such as pixel kernel 25 is used forpixel G_(3,3). Thus, as pixel kernel 25 moves across and down a pixelarray, green pixels at the center of the pixel kernel 25 are corrected.

It should be understood that this method is but one example of a localgreen imbalance compensation method. This type of estimation method forlocal green imbalance compensation works well in areas of an image thatare uniform. However, the presence of sharp, high color-contrast edgesmay introduce large errors in the estimated correction term, Δ_(i,j).These errors become apparent in the corrected image as zipper effects orcolor fringe around the high-contrast edges. Other types of imageartifacts may also occur while using this type of method for local greenimbalance estimation and correction.

Disclosed embodiments improve upon this type of average-based localgreen imbalance compensation method by limiting the correction term,Δ_(i,j), within predetermined limits in order to minimize unwanted imageartifacts. Requiring the correction term to fall within thepredetermined limits (determined during imager pixel array and/or cameracalibration), reduces the effect of high-contrast edges on the localgreen imbalance correction algorithm. Disclosed embodiments of limitingthe green balance correction term may also be implemented using othermethods of calculating the green balance correction term, Δ_(i,j), thanthose described herein.

According to disclosed embodiments, the estimated local green balancecorrection term, Δ_(i,j), is limited as shown in the following Equations(7) through (9):Δ′_(i,j) =a _(pos) *p _(i,j) +b _(pos) when Δ_(i,j)>a_(pos) *p _(i,j) +b_(pos);   (7)Δ′_(i,j) =a _(neg) *p _(i,j) +b _(neg) when Δ_(i,j) <a _(neg) *p _(i,j)+b _(neg); and   (8)Δ′_(i,j)=Δ_(i,j) when a_(neg) *p _(i,j) +b _(neg)≦Δ_(i,j) ≦a _(pos) *p_(i,j) +b _(pos),   (9)where Δ_(i,j) represents the estimated local green balance correctionterm (determined from any available green imbalance compensation method,such as that previously discussed), Δ′_(i,j) represents the limitedgreen balance correction term, and a_(neg), b_(neg), a_(pos), andb_(pos) are constants. The constants depend on various properties of thepixel being corrected and on the sensor and camera optics by which theimage is captured. The constants are determined during imager and/orcamera calibration, discussed in more detail below. It should be notedthat the correction term is limited to be between the upper and lowerlimits, as shown in Equation (10):Δ_(neg) _(—) _(i,j) =a _(neg) *b _(neg)≦Δ′_(i,j) ≦a _(pos) *p _(i,j) +b_(pos)=Δ_(pos) _(—) _(i,j).   (10)where Δ_(neg) _(—) _(i,j) is the negative limit and Δ_(pos) _(—) _(i,j)is the positive limit on the green balance correction term. The positiveand negative limits, Δ_(pos) _(—) _(i,j) and Δ_(neg) _(—) _(i,j),respectively, are derived from the calculated parameters determinedduring imager pixel array and/or camera calibration.

When the estimated local green balance correction term, Δ_(i,j), isoutside of these limits, Δ_(pos) _(—) _(i,j) and Δ_(neg) _(—) _(i,j),the limits are applied to determine a limited correction term, Δ′_(i,j).If the estimated local green balance correction term already satisfiesthe relationship of Equation (10) (e.g., Δ_(i,j) is already between thelimits Δ_(pos) _(—) _(i,j) and Δ_(neg) _(—) _(i,j)), then no limiting ofthe correction term is needed and the limited green balance correctionterm is equal to the estimated local green balance correction term.Disclosed embodiments then substitute the limited green balancecorrection term, Δ′_(i,j), for the estimated local green balancecorrection term, Δ_(i,j), when using Equations (4) and (5) to calculatethe corrected pixel signal and complete green imbalance compensation.The limited green balance correction term, Δ′_(i,j), is calculated inthis manner for all green pixels of the pixel array.

The correction term limits, Δ_(pos) _(—) _(i,j) and Δ_(neg) _(—) _(i,j),applied in Equation (10) are derived as follows. In general, the amountof pixel crosstalk is a function of exposure and spectral reflectance ofthe object. In particular, the optical crosstalk is typically a linearfunction of exposure, which in turn is linearly related to pixelresponse. Thus, assuming the optical crosstalk component substantiallydominates the electrical crosstalk component, the pixel crosstalk is alinear function of pixel response. As previously discussed, greenimbalance may be caused by optical or electrical crosstalk among pixels.A camera imaging a white flat field illuminated by a light source havinga certain spectral characteristic results in a pixel response that islinear, regardless of the spectral characteristic (S). The pixelresponse can be thought of as a sum of two components—exposure withoutcrosstalk and inter-pixel crosstalk. By pixel array design, thenon-crosstalk component is linear with respect to exposure. If crosstalktakes place, the amount of the color crosstalk component thereforeshould also be a linear function of exposure. The slope of the crosstalkcomponent (a_(neg), a_(pos)) depends on the particular choice of thespectral characteristic of the illuminant and the spectral reflectanceof the object being imaged. Thus, in any real scene, the amount of greenimbalance is a linear function of exposure with the slope depending onthe spectrum of the light reflected from the object being imaged.

As seen in FIG. 3, for a given pixel, a maximum positive limit 61,a_(pos) *p _(i,j), and a maximum negative limit 62, a_(neg) *p _(i,j),can be found such that all dependencies Δ_(i,j)=ƒ(p_(i,j)) comply witha_(neg)*p_(i,j)≦Δ_(i,j)<a_(pos)*p_(i,j) for all spectra of lightreflected from the imaged object. In theory, these limits improve thegreen imbalance adjustment. However, in a practical application, thepixel response may not exactly be a linear function of exposure. Thus, asomewhat more general formula, as shown in Equation (10) above, thatincludes offsets (b_(neg), b_(pos)) chosen to account for possible smallnon-linearities is used.

Parameters a_(neg), b_(neg), a_(pos), and b_(pos) are determined duringa calibration process for an imager pixel array or for a camera usingthe array. This can be done either experimentally or using a simulation.The amount of pixel crosstalk affecting pixels in a given imager arraymay or may not depend on the location of the specific pixel within thepixel array. For example, a pixel array may only experience electricalcrosstalk, which would not be a function of pixel location within thearray. If pixel crosstalk is not a function of pixel location in thepixel array, a single set of parameters (a_(neg), b_(neg), a_(pos), andb_(pos)) is determined. If pixel crosstalk is a function of pixellocation in the pixel array, a plurality of sets of parameters, one foreach pixel location (a_(neg) _(—) _(i,j), b_(neg) _(—) _(i,j), a_(pos)_(—) _(i,j), and b_(pos) _(—) _(i,j) ), are determined. More often thannot, pixel crosstalk is a function of the location of the pixel withinthe pixel array, so a plurality of sets of parameters are used.

A calibration process for an imager wherein the amount of pixelcrosstalk is not a function of pixel location in the pixel array isillustrated, in flowchart form, in FIG. 4. This calibration processdetermines a single set of parameters for use in green balancecorrection term limiting. At step S10, the camera exposure is set to afixed value. At step S20, the imager (e.g., a camera) images a testobject having all possible spectral reflectances that the designerexpects the camera to encounter and image during use. In a typicalexperiment, the test object would be the known Macbeth color chart; theversion for digital cameras having a large collection of patches ispreferably used. At step S20, the images are taken using a set ofilluminants covering the expected color temperature range of operation,for example at 6500K, 4100K, 3100K and 2800K. In this calibrationtechnique, where it is assumed that pixel location does not effectcrosstalk, the placement of the chart is irrelevant because pixelcrosstalk is not considered to be a function of pixel location in thearray.

The illumination of the Macbeth color chart is increased and theestimated local green balance correction term, Δ_(i,j), is determinedusing Equations (1) through (3) at step S30 for each spectral response.The spectral response describes the sensitivity of the photosensor tooptical radiation of different wavelengths, i.e., illuminant colortemperatures. Then, at step S40, the values of the estimated local greenbalance correction term, Δ_(i,j), are plotted against pixel response(calculated as an average of a small local neighborhood) and a linearfunction is fit to the plotted data. At step S50, the slope, a_(i,j),and offset, b_(i,j), of the linear function are calculated for eachspectral response. For the Macbeth chart, the set of spectral responsesis determined by the product of the set of patches on the Macbeth chartand the set of illuminants used to take test images. At step S60, theparameters, a_(neg), b_(neg), a_(pos), and b_(pos), are determined, fromthe set of linear functions, by determining positive and negative limitfunctions. The positive limit function may be a linear functionƒ_(pos)(p)=a_(pos)*p+b_(pos) such that ƒ_(pos)(p) always equals orexceeds values of all estimated positive limit functions for all pixelvalues p. Similarly, the negative function is a linear functionƒ_(neg)(p)=a_(neg)*p+b_(neg) such that ƒ_(neg)(p) is always less than orequal to all estimated negative limit functions for all pixel values p.The parameters are calculated as shown in Equations (11) and (12),respectively:a _(neg) and b _(neg), such that a _(neg) *p+b _(neg) ≦a _(neg) _(—)_(i,j) *p+b _(neg) _(—) _(i,j) for all values of i,j and p, whileminimizing the overall aggressiveness of green imbalance correction by,for example, minimizing sum(|a _(neg) *p+b _(neg)|) for all p;   (11)a _(pos) and b _(pos) are minimum, such that a_(pos) *p+b _(pos) ≧a_(pos) _(—) _(i,j) *p+b _(pos) _(—) _(i,j) for all values of i, j and p,while minimizing the overall aggressiveness of green imbalancecorrection by, for example, minimizing sum(|a _(pos) *p+b _(pos)|) forall p.   (12)The determined parameters are then stored at step S70 for later use indetermining the limits on the green balance compensation terms duringimage processing. Alternatively, other parameters for defining piecewisenon-linear equations from which the determined parameters can becalibrated could be stored.

A calibration process for an imager pixel array wherein the amount ofpixel crosstalk is a function of pixel location in the pixel array isillustrated, in flowchart form, in FIG. 5. This calibration processdetermines a set of parameters (a_(neg) _(—) _(i,j), b_(neg) _(—)_(i,j), a_(pos) _(—) _(i,j), and b_(pos) _(—) _(i,j)) for each pixellocation (i, j) for use in limiting a green balance correction term. Thesteps with the same reference numerals in FIG. 5 are performed in thesame manner as described with reference to FIG. 4 above. FIG. 5 includesthe additional steps of moving the test object to a new locationrelative to the scene imaged by the imager (step S18) and repeatingsteps S18 through S20 until the last location has been imaged (stepS24). The test object, e.g., the Macbeth color chart, is placed in avariety of locations to be imaged, such as corners, sides and imagercenter. A plurality of sets of parameters (a_(neg) _(—) _(i,j), b_(neg)_(—) _(i,j), a_(pos) _(—) _(i,j), and b_(pos) _(—) _(i,j)) aredetermined at step S61 (using Equations (11) and (12)), with one setbeing calculated for each of the locations at which the Macbeth colorchart is placed.

The parameters are stored at step S70 for later use during imageprocessing. Alternatively, parameters can be stored which are capable ofgenerating piecewise non-linear equations which define the greenimbalance correction parameters can be stored. In theory, forimplementations in which the pixel location affects the green imbalancecorrection, a set of parameters can be determined and stored for eachgreen pixel location in the pixel array. However, in practicalimplementations, the parameters may be stored in the form of a tablespecifying values only in several points, for example, for arraycorners, sides and center. Values for remaining locations may then bedetermined using bi-linear interpolation.

Alternatively, the parameters could be set to a constant equal to theirmaximum positive and negative values across the entire array tosubstantially simplify implementation. However, using this method willresult in the correction terms not being limited in an optimal way incertain areas in the array.

The parameters could also be set using a combination of these methods. Awindow is defined which covers the image center (x₀, y₀, x₁, y₁). Withinthis window, the parameters, a_(neg), b_(neg), a_(pos), and b_(pos), areconstant. Outside of the window (e.g., toward the image periphery), theparameters will increase as a function of the distance from the window,e.g., a_(pos) =a _(pos)+depth pos (dx, dy). This method of parametersetting is based on the observation that green imbalance is typicallystrongest on the image periphery, and therefore more correction isrequired at those locations. Two methods of calculation of da_(pos)(dx,dy) are shown in Equations (13) and (14). It should be noted that thesame approach is taken with the other three parameters, a_(neg),b_(neg), and b_(pos).da _(pos)(dx,dy)=da _(pos) _(—) _(x) *dx+da _(pos) _(—) _(y) *dy.   (13)da _(pos)(dx,dy)=max(da _(pos) _(—) _(x) *dx+da _(pos) _(—) _(y) *dy).  (14)where dx=x₀−x when x<x₀; dx=x−x₁ when x>x₁; dy=y₀−y when y<y₀; dy=y−y₁when y>y₁; for each of Equations (13) and (14).

Once the imager has been calibrated, the stored parameters are usedduring image processing to implement green imbalance compensation on thegreen pixels of the captured image. FIG. 6 shows a method of greenimbalance compensation in accordance with disclosed embodiments. FIG. 7illustrates a method for determining the positive and negative limits,Δ_(pos) and Δ_(neg), of the correction term. FIG. 8 illustrates a systemthat implements the method of FIGS. 6 and 7.

Referring to FIG. 6, at step S80, the estimated local green balancecorrection term, Δ_(i,j), is determined in accordance with any knownmethod. Then at step S90, the upper and lower limits, Δ_(pos) _(—)_(i,j) and Δ_(neg) _(—) _(i,j), of the correction term are determined.It is important to note that the limited correction term may be the sameas the estimated local green balance correction term when the localestimated green balance correction term is already within thepredetermined limits, as previously discussed. At step S100, the limitedgreen balance correction term, Δ′_(i,j), is determined. Then, at stepS110, limited green balance correction term, Δ′_(i,j), is applied to thepixel signal. This may be done in the limiter 154 of FIG. 8.

In the method of FIG. 7, a plurality of parameter sets have beendetermined and stored for various locations throughout the array, asdiscussed above with reference to FIG. 5. At step S120, the pixellocation (i, j) for a green pixel for which crosstalk is to be correctedis input into a limits calculator 150 (FIG. 8). Then, the four nearestavailable values of the four parameters (a_(neg) _(—) _(i,j), b_(neg)_(—) _(i,j), a_(pos) _(—) _(i,j), and b_(pos) _(—) _(i,j)) surroundingthe pixel location are found in the stored tables of parameters 151 atstep S130. At step S140, a bi-linear interpolator 152 determines thevalues of the parameters for the pixel location (i,j). At step S150,Δ_(pos) _(—) _(i,j) and Δ_(neg) _(—) _(i,j) are determined in a Δ_(pos),Δ_(neg) calculator 153 using the parameters determined at step S140 andEquation (10). In this embodiment, a_(neg) is replaced by a_(neg) _(—)_(i,j), etc. in Equation (10). Parameters a_(neg) _(—) _(i,j), b_(neg)_(—) _(i,j), a_(pos) _(—) _(i,j), and b_(pos) _(—) _(i,j) have beencalibrated for a plurality of pixel locations throughout the array, inaccordance with FIG. 5.

FIG. 8 illustrates a hardware implementation of position-dependentcorrection term limits that includes a limits calculator 150 forcalculating position-dependent parameters a_(neg) _(—) _(i,j), b_(neg)_(—) _(i,j), a_(pos) _(—) _(i,j), and b_(pos) _(—) _(i,j). Theparameters determined during imager pixel array calibration (FIGS. 4 and5) may be stored in table 151. The table has entries denoting values ofcorresponding parameters as a grid of size K by L, where K is much lessthan M and L is much less than N. For example, the parameters can bespecified for points on a coarse 5×5 grid overlaid on the pixel arrayinstead of specifying values for each pixel, p_(i,j). The parameters foruse in limiting the green balance correction term for pixel locationsfor which the corresponding parameters are not stored may then becalculated by bi-linear interpolation as described with reference toFIG. 7.

Alternatively, the limits, Δ_(pos) _(—) _(i,j) and Δ_(neg) _(—) _(i,j)of the correction term may be determined using parameters from a look-uptable without interpolation or from a single set of parameters (asdetermined with reference to FIG. 4) or from stored parameters which canbe used to generate piecewise non-linear equations which can generateparameters which are in turn used to generate the limits Δ_(pos) _(—)_(i,j), and Δ_(neg) _(—) _(i,j).

FIG. 9 illustrates a block diagram of a system-on-a-chip (SOC) imager100 which may use any type of image sensing technology, CCD, CMOS, etcand which implements the previously described method.

The imager 100 comprises a sensor core 200 that communicates with animage processor circuit 110 that is connected to an output interface130. A phase lock loop (PLL) 244 is used as a clock for the sensor core200. The image processor circuit 110, which is responsible for image andcolor processing, includes interpolation line buffers 112, decimatorline buffers 114, and a color processing pipeline 120. The colorprocessing pipeline 120 includes, among other things, a limitscalculator 150 and limiter 154, shown in FIG. 8. One of the functions ofthe color processing pipeline 120 is the performance of green imbalancecompensation in accordance with the embodiments discussed above. Theimage processor circuit 110 may also be implemented as a DSP or a hostcomputer.

The output interface 130 includes an output first-in-first-out (FIFO)parallel buffer 132 and a serial Mobile Industry Processing Interface(MIPI) output 134, particularly where the imager 100 is used in a camerain a mobile telephone environment. The user can select either a serialoutput or a parallel output by setting registers in a configurationregister within the imager 100 chip. An internal bus 140 connects readonly memory (ROM) 142, a microcontroller 144, and a static random accessmemory (SRAM) 146 to the sensor core 200, image processor circuit 110,and output interface 130. The read only memory (ROM) 142 or SRAM 146 mayserve as a storage location for one or more sets of stored parametervalues (e.g., a_(neg) _(—) _(i,j), b_(neg) _(—) _(i,j), a_(pos) _(—)_(i,j), and b_(pos) _(—) _(i,j)), or for parameters describing apiecewise non-linear equation from which these values can be derived.

FIG. 10 illustrates a sensor core 200 that may be used in the imager 100(FIG. 9). The sensor core 200 includes, in one embodiment, a pixel array202. Pixel array 202 is connected to analog processing circuit 208 by agreen1/green2 channel 204, which outputs pixel output signalscorresponding to two green channels of the pixel array 202, and througha red/blue channel 206, which contains pixel output signalscorresponding to the red and blue channels of the pixel array 202.

Although only two channels 204, 206 are illustrated, there areeffectively two green channels, green-red and green-blue, as well as redand blue channels. The green1 (i.e., green-red) and green2 (i.e.,green-blue) signals are read out at different times (using channel 204)and the red and blue signals are read out at different times (usingchannel 206). The analog processing circuit 208 outputs processedgreen1/green2 signals G1/G2 to a first analog-to-digital converter (ADC)214 and processed red/blue signals R/B to a second analog-to-digitalconverter 216. The outputs of the two analog-to-digital converters 214,216 are sent to a digital processing circuit 230.

Connected to, or as part of, the pixel array 202 are row and columndecoders 211, 209 and row and column driver circuitry 212, 210 that arecontrolled by a timing and control circuit 240 to capture images usingthe pixel array 202. The timing and control circuit 240 uses controlregisters 242 to determine how the pixel array 202 and other componentsare controlled. As set forth above, the PLL 244 serves as a clock forthe components in the sensor core 200.

The pixel array 202 comprises a plurality of pixels arranged in apredetermined number of columns and rows. For a CMOS imager, the pixelsof each row in the pixel array 202 are all turned on at the same time bya row select line and the pixels of each column within the row areselectively output onto column output lines by a column select line. Aplurality of row and column lines are provided for the entire pixelarray 202. The row select lines are selectively activated by row drivercircuitry 212 in response to row decoder 211 and column select lines areselectively activated by a column driver 210 in response to columndecoder 209. Thus, a row and column address is provided for each pixel.The timing and control circuit 240 controls the row and column decoders211, 209 for selecting the appropriate row and column lines for pixelreadout, and the row and column driver circuitry 212, 210, which applydriving voltage to the drive transistors of the selected row and columnlines.

Each column contains sampling capacitors and switches in the analogprocessing circuit 208 that read a pixel reset signal Vrst and a pixelimage signal Vsig for selected pixels. Because the sensor core 200 usesa green1/green2 channel 204 and a separate red/blue channel 206, analogprocessing circuit 208 will have the capacity to store Vrst and Vsigsignals for green1/green2 and red/blue pixel output signals. Adifferential signal (Vrst−Vsig) is produced by differential amplifierscontained in the analog processing circuit 208. This differential signal(Vrst−Vsig) is produced for each pixel output signal. Thus, the signalsG1/G2 and R/B are differential signals representing respective pixeloutput signal values that are digitized by a respectiveanalog-to-digital converter 214, 216. The analog-to-digital converters214, 216 supply the digitized G1/G2 and R/B pixel output signals to thedigital processing circuit 230 which forms the digital image output (forexample, a 10 bit digital output). The output is sent to the imageprocessor circuit 110 (FIG. 9) for further processing. The imageprocessor circuit 110 will, among other things, perform a greenimbalance compensation on the digital pixel output signal values of thecaptured image. Although a CMOS sensor core has been described forproviding pixels output signals of a captured image, other solid stateimager sensor core architectures and readout devices, such as CCD andothers, may also be used.

The color processing pipeline 120 of the image processor circuit 110performs a number of operations on the pixel output signals receivedthereat. In accordance with embodiments described herein, the greenimbalance compensation may be performed using a set of stored parametersavailable in, for example, ROM 142, SRAM 146 or other forms of storage(e.g., registers).

A software implementation of determining the correction term limits mayalso be used. The software implementation would be functionallyidentical to the hardware implementation except that it is written onexecutable code stored on a memory medium and executed by a processor.

As previously discussed, embodiments may be implemented as part of apixel output signal processing pipeline 110 by a processor executing aprogram, by hardware circuits within processing pipeline 110, or by acombination of the two, or by a stand-alone processing system.

Embodiments may be implemented as part of a camera such as e.g., adigital still or video camera, or other image acquisition system, andalso may be implemented as a stand-alone or plug-in software componentfor use in image editing applications. In such applications, the processdescribed above with reference to FIGS. 4-7 can be implemented ascomputer instruction code stored as a set of instructions on a storagemedium and executable on a processor. The stored parameters arecontained on a storage medium for use in a computer image processingsystem.

FIG. 11 illustrates a processor system as part of, for example, ordigital still or video camera system 800 employing a system-on-a-chipimager 100 as illustrated in FIG. 8, which imager 100 provides for greenimbalance compensation and other pixel output signal corrections asdescribed above. The processing system includes a processor 805 (shownas a CPU) which implements system, e.g. camera 800, functions and alsocontrols image flow. The processor 805 is coupled with other elements ofthe system, including random access memory 820, removable memory 825such as a flash or disc memory, one or more input/out devices 810 forentering data or displaying data and/or images and imager 100 throughbus 815 which may be one or more busses or bridges linking the processorsystem components. Camera 800 further includes a lens 835 that allowslight to pass to the imager 100 when shutter release button 840 isactivated.

The camera system 800 is an example of a processing system havingdigital circuits that could include imagers. Without being limiting,such a system could include a computer system, camera system, scanner,machine vision, vehicle navigation, video phone, surveillance system,auto focus system, star tracker system, motion detection system, imagestabilization system, and after image processing system.

While described embodiments have been described in detail, it should bereadily understood that the invention is not limited to the disclosedembodiments. Rather the embodiments can be modified to incorporate anynumber of variations, alterations, substitutions or equivalentarrangements not heretofore described.

1. A method of processing a plurality of pixel signals corresponding toan image comprising: determining an estimated green balance correctionterm for each green pixel signal; determining a limited green balancecorrection term for each green pixel signal by limiting the estimatedgreen balance correction term based on a set of parameters; andadjusting each green pixel signal by applying the limited green balancecorrection term to the green pixel signal.
 2. The method of claim 1,wherein limiting the estimated green balance correction term furthercomprises applying at least one of an upper limit and a lower limit tothe estimated green balance correction term, wherein the upper and lowerlimits are determined from the set of parameters.
 3. The method of claim1 wherein the set of parameters is obtained from a storage location thatincludes a plurality of sets of parameters, respectively associated withdifferent ones of the plurality of green pixel signals.
 4. The method ofclaim 1 wherein determining the estimated green balance correction termfurther comprises: determining a first average of pixel signals in apixel kernel surrounding a particular green pixel being processed andbelong to a first green color channel, wherein the particular pixelbelongs to the first green color channel; determining a second averageof pixel signals in a pixel kernel surrounding the particular greenpixel and belong to a second green color channel; and determining theestimated green balance correction term based on a difference betweenthe first average and the second average.
 5. The method of claim 4,wherein the estimated green balance correction term is determined as${\Delta_{i,j} = \frac{S_{G\; 1} - S_{G\; 2}}{2}},$ wherein S_(G1) isthe first average and S_(G2) is the second average.
 6. The method ofclaim 2, wherein the set of parameters comprises a_(pos), b_(pos),a_(neg), and b_(neg) and are stored in a storage location, whereina_(pos) and a_(neg) are the slope and b_(pos), and b_(neg) are theintercepts of functions determined during a calibration procedure andthe parameters are related to at least one of the particular pixelsignal, a sensor which captures the image and camera optics of a camerawhich captures the image, wherein the upper and lower limits aredetermined by applying the set of parameters as follows:Δ_(pos) =a _(pos) *p _(i,j) +b _(pos); andΔ_(neg) =a _(neg) *p _(i,j) +b _(neg), and wherein Δ_(pos) is the upperlimit, Δ_(neg) is the lower limit and p_(i,j) is a particular pixelsignal.
 7. The method of claim 4, wherein adjusting the pixel signalsfurther comprises applying the limited local green balance correctionterm to each green pixel signal as follows: p′_(i,j)=p_(i,j)−|Δ_(i,j)|,when the first average is greater than the second average;p′_(i,j)=p_(i,j)+|Δ_(i,j)|, when the first average is less than thesecond average; and p′_(i,j)=p_(i,j), when the first average is equal tothe second average, wherein p′_(i,j) is the adjusted pixel signal,p_(i,j) is a particular pixel signal, Δ_(i,j) is the limited local greenbalance correction term, and i and j represent a location of theparticular green pixel within a pixel array.
 8. The method of claim 1wherein the set of parameters is related to at least one of an arraylocation of a particular green pixel signal, a sensor which captures theimage and camera optics of a camera which captures the image.
 9. Themethod of claim 1, wherein the set of parameters is obtained by:obtaining from a storage location, for each parameter, a set ofavailable values associated with an array location nearest a particulargreen pixel being adjusted; and using interpolation to determine eachparameter for the particular green pixel being adjusted.
 10. The methodof claim 9 wherein the interpolation is a bi-linear interpolation.
 11. Amethod of operating an imaging system, the method comprising: acquiringa plurality of test images; developing parameters for limiting greenimbalance color correction terms from the test images; and storing theparameters in an imager for use in correction of green color imbalanceby the imager.
 12. The method of claim 11, wherein the plurality of testimages are acquired for at least one of a respective plurality ofdifferent color temperature illumination conditions and a respectiveplurality of locations relative to an imager array.
 13. The method ofclaim 11 wherein the plurality of test images are acquired of a testobject including a plurality of colors.
 14. The method of claim 11,wherein developing the parameters further comprises: determining anestimated local green balance correction term for each test image;determining data points for the estimated local green balance correctionterm with respect to a pixel response for each test image; fitting alinear function to the data points for each test image; determining setsof possible parameters for each test image; and choosing the appropriateset of parameters for use in correction term limiting for each testimage.
 15. The method of claim 14, wherein determining sets of possibleparameters for each test image comprises determining a slope and anintercept of the linear function fitted to the data points for each testimage.
 16. The method of claim 14, wherein choosing the appropriate setof possible parameters for use in correction term limiting compriseschoosing the sets of possible parameters which yield the largestpositive and the largest negative green imbalance estimate.
 17. Themethod of claim 11, further comprising storing a plurality ofappropriate sets of parameters for use in correction term limiting,wherein each of the plurality of sets of parameters relates to aparticular pixel location within a pixel array.
 18. An imaging devicecomprising: an array of pixels for capturing an image and providingpixel signals; a storage device for storing at least one set ofparameters; and a processing circuit for processing pixel output signalsproduced by the array and corresponding to a captured image, theprocessing circuit being configured to: determine an estimated greenbalance correction term for each green pixel output signal; determine alimited green balance correction term for each green pixel output signalby limiting the estimated green balance correction term by applying atleast one of an upper limit and a lower limit to the estimated greenbalance correction term, wherein the upper and lower limits aredetermined from the at least one set of parameters; and adjust eachgreen pixel output signal by applying the limited green balancecorrection term to the green pixel signal.
 19. The imaging device ofclaim 18, wherein the set of parameters comprises a_(pos), b_(pos),a_(neg), and b_(neg) and are stored in a storage location, whereina_(pos) and a_(neg) are the slope and b_(pos), and b_(neg) are theintercepts of functions determined during a calibration procedure andthe parameters are related to at least one of the particular green pixelsignal, a sensor which captures the image and camera optics of a camerawhich captures the image, wherein determining the upper and lower limitscomprises applying the set of parameters as follows:Δ_(pos) =a _(pos) *p _(i,j) +b _(pos); andΔ_(neg) =a _(neg) *p _(i,j) +b _(neg), and wherein Δ_(pos) is the upperlimit and Δ_(neg) is the lower limit.
 20. The imaging device of claim18, wherein determining the estimated green balance correction termfurther comprises: determining a first average of pixel output signalsin a pixel kernel surrounding a particular green pixel being processedand belong to a first green color channel, wherein the particular greenpixel belongs to the first green color channel; determining a secondaverage of pixel output signals in a pixel kernel surrounding theparticular green pixel and belong to a second green color channel; anddetermining the estimated green balance correction term based on adifference between the first average and the second average.
 21. Theimaging device of claim 20, wherein adjusting the pixel output signalsfurther comprises applying the limited green balance correction term tothe green pixel output signal as follows: p′_(i,j)=p_(i,j)−|Δ_(i,j)|,when the first average is greater than the second average;p′_(i,j)=p_(i,j)+|Δ_(i,j)|, when the first average is less than thesecond average; and p′_(i,j)=p_(i,j), when the first average is equal tothe second average, wherein p′_(i,j) is the adjusted pixel signal,p_(ij) is a particular green pixel output signal, Δ,_(ij) is the limitedlocal green balance correction term, and i and j represent a location ofthe particular green pixel within the array of pixels.
 22. The imagingdevice of claim 18, wherein the set of parameters is obtained by:obtaining from a storage location, for each parameter, a set ofavailable values associated with an array location nearest a particulargreen pixel being adjusted; and using interpolation to determine the setof parameters for the green particular pixel being adjusted.
 23. Adigital camera comprising: a pixel array for capturing an image receivedthrough a lens; a storage area for storing at, least one set ofparameters; and a pixel array processing circuit which, using the atleast one set of parameters, is configured to determine a limited greenbalance correction term, and to adjust green pixel signals for acaptured image using the limited green balance correction term, whereinthe limited green balance correction term is determined by limiting anestimated green balance correction term using the at least one set ofparameters, and wherein the estimated green balance correction term islimited such that the limited green balance correction term is greaterthan or equal to a lower limit and less than or equal to an upper limit,wherein the lower and upper limits are determined from the at least oneset of parameters.
 24. The digital camera of claim 23, wherein the setof parameters comprises a_(pos), b_(pos), a_(neg), and b_(neg) and isstored in a storage location, wherein a_(pos) and a_(neg) are the slopeand b_(pos), and b_(neg) are the intercepts of functions determinedduring a calibration procedure and the parameters are related to atleast one of the particular green pixel signal, a sensor which capturesthe image and camera optics of a camera which captures the image,wherein the upper and lower limits are determined by applying the set ofparameters as follows:Δ_(pos) =a _(pos) *p _(i,j) +b _(pos); andΔ_(neg) =a _(neg) *p _(i,j) +b _(neg), wherein Δ_(pos) is the upperlimit, Δ_(neg) is the lower limit and P_(i,j) is the green pixel signal.25. A non-transitory computer readable medium having stored thereon acomputer program comprising a set of instructions executed by a computerto implement a method for green imbalance correction, the methodcomprising the acts of: determining an estimated green balancecorrection term for each green pixel signal associated with a capturedimage, wherein determining the estimated green balance correction termcomprises: determining a first average of pixel signals in a pixelkernel surrounding a particular green pixel being processed and belongto a first green color channel, wherein the particular pixel belongs tothe first green color channel; determining a second average of pixelsignals in a pixel kernel surrounding the particular green pixel andbelong to a second green color channel; and determining the estimatedgreen balance correction term based on a difference between the firstaverage and the second average; determining a limited green balancecorrection term for each green pixel signal by limiting the estimatedgreen balance correction term by applying at least one of an upper limitand a lower limit to the estimated green balance correction term,wherein the upper and lower limits are determined from a set ofparameters, the set of parameters being related to at least one of anarray location of a particular green pixel signal, a sensor whichcaptures the image and camera optics of a camera which captures theimage, and the set of parameters being obtained by: obtaining from astorage location, for each parameter, a set of available valuesassociated with an array location nearest the particular green pixelbeing adjusted; and using interpolation to determine the set ofparameters for the green particular pixel being adjusted; and adjustingeach green pixel signal by applying the limited green balance correctionterm to the green pixel signal.